Calibration method, calibration system, and program

ABSTRACT

This method is for calibrating a coordinate system of an image capture device and a coordinate system of a robot arm in a robot system that includes a display device, the image capture device, and the robot arm to which one of the display device and the image capture device is fixed, the robot arm having a drive shaft. The method includes: acquiring first captured image data based on first image data; acquiring second captured image data based on second image data different from the first image data; and calibrating the coordinate system of the image capture device and the coordinate system of the robot arm, using the first captured image data and the second captured image data.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims priority to Japanese Patent Application No.2017-154016 filed Aug. 9, 2017, the entire contents of which areincorporated herein by reference.

FIELD

The disclosure relates to a calibration method, a calibration system,and a program.

BACKGROUND

As a method for performing a calibration of relative positionalorientations between a coordinate system of a robot and a coordinatesystem of an image sensor, a method is conventionally known in which animage of a flat plate (calibration object) with dots and a polygonalmark marker printed thereon is captured by an image sensor, andcalibration computation is performed based on the captured image.

For example, Patent Document 1 describes a calibration method that aimsto reduce calibration errors for an articulated robot that hashysteresis characteristics in its operation. Specifically, in thismethod, a coordinate system of a robot and a coordinate system of animage sensor are calibrated using a calibration object. An image iscaptured at a first measurement point using an image capture device, andwhen the orientation of the robot is changed to reach a secondmeasurement point at which the robot assumes a different orientation,the robot is caused to reach the second measurement point via apredetermined specific point.

JP 2015-182144 is an example of background art.

When a calibration of a positional relationship between a coordinatesystem of a robot and a coordinate system of an image capture device isperformed using a calibration object, computation for the calibration isperformed based on a captured image of the calibration object acquiredat respective measurement points by the image capture device.Accordingly, it is favorable that the mode of calibration object isappropriately selected in accordance with measurement conditions.

However, the calibration method disclosed in Patent Document 1 uses asingle calibration object (print) with a plurality of dots and a markmarker that are predetermined printed thereon. For this reason, thereare cases where the calibration accuracy cannot be ensured depending onthe captured image. To ensure the calibration accuracy, prints ofcalibration objects with a plurality of patterns printed thereon need tobe prepared, and thus the number of man-hours of calibration increases.

One or more aspects may provide a calibration method and system, and aprogram that enable a calibration of a relative positional relationshipbetween an image capture device and a robot arm without preparingcalibration objects with a plurality of patterns printed thereon.

SUMMARY

A calibration method according to an aspect is a method for calibratinga coordinate system of an image capture device and a coordinate systemof a robot arm in a robot system that includes a display device, theimage capture device, and the robot arm to which one of the displaydevice and the image capture device is fixed, the robot arm having adrive shaft. This calibration method includes a step of causing thedisplay device to display a first image based on first image data; astep of capturing an image of the first image displayed on the displaydevice and acquiring first captured image data, using the image capturedevice; a step of causing the display device to display a second imagebased on second image data different from the first image data; a stepof capturing an image of the second image displayed on the displaydevice and acquiring second captured image data, using the image capturedevice; and a step of calibrating the coordinate system of the imagecapture device and the coordinate system of the robot arm using thefirst captured image data and the second captured image data.

According to an aspect, a calibration of a relative positionalrelationship between the image capture device and the robot arm can beexecuted without preparing calibration objects with a plurality ofpatterns printed thereon. Since a calibration can be performed using aplurality of captured images in different forms of display, thecalibration accuracy can be improved.

In an aspect, data indicating a pattern in an image to be displayed ordata indicating a brightness of the image to be displayed may differbetween the first image data and the second image data.

According to an aspect, an image pattern with a brightness appropriatefor calibration can be displayed in accordance with the brightness ofthe surroundings or the like. For example, there are cases wherebrightness conditions differ between measurement points for measuringthe image using the image capture device, in accordance with an angle,illuminance, or the like of lighting, and cases where brightnessconditions vary even at the same measurement point. Accordingly,according to an aspect, an appropriate image pattern can be displayed inaccordance with different brightness conditions, and thus thecalibration accuracy can be improved.

Furthermore, according to an aspect, the pattern in the image to bedisplayed can be differentiated, and thus the calibration accuracy canbe improved. That is to say, since a calibration can be performed usinga plurality of captured images of different patterns, the calibrationaccuracy can be improved. In addition, since the pattern can bedifferentiated in accordance with an environment in which an image is tobe captured, the calibration accuracy can be improved.

In an aspect, each of the first image data and the second image data maybe data indicating a pattern in an image to be displayed, and dataregarding a color of at least one pixel in the image to be displayed onthe display device may differ between the first image data and thesecond image data.

According to an aspect, the information regarding the captured imagedata can be increased, and the calibration accuracy can be improved.

In an aspect, the calibration method may further include a step ofchanging relative positions of the display device and the image capturedevice using the robot arm, after acquiring the first captured imagedata, and the step of acquiring the second captured image data may beperformed after the step of changing the relative positions.

According to an aspect, images that are displayed based on differentimage data are captured with different relative positions, and thecaptured image data is acquired. Thus, the calibration accuracy can beimproved. For example, a configuration is employed in which differentcaptured image data is acquired to perform calibration computationbefore and after the robot arm is operated to a measurement point atwhich the relative positions of the display device and the image capturedevice differ. Thus, an image appropriate for measurement conditions,such as a relative positional relationship between the display deviceand the image capture device, can be selected, and the calibrationaccuracy can be improved.

In an aspect, during the step of causing the display device to displaythe first image and the step of causing the display device to displaythe second image, in the first image data or the second image data, asize of a pattern in the first image or the second image to be displayedmay be changed in accordance with coordinates of a leading end of therobot arm.

According to an aspect, the image data is changed so as to differentiatethe size of the image pattern in accordance with the coordinates of theleading end of the robot arm, and thus, an image pattern with a sizeappropriate for calibration can be displayed in accordance with arelative positional relationship between the display device and theimage capture device. A calibration of a coordinate system of an imagecapture device and a coordinate system of a robot arm is performed toimprove the accuracy of predetermined processing for an object using arobot arm (e.g. gripping, suction, fitting, winding etc. of the object).Accordingly, the calibration accuracy can be improved by changing theimage data so as to differentiate the size of the image pattern inaccordance with the coordinates of the leading end of the robot arm thatacts on an object, and performing a calibration using a plurality oftypes of captured images that are based on different image patterns.

In an aspect, during the step of causing the display device to displaythe first image and the step of causing the display device to displaythe second image, in the first image data or the second image data, asize of a pattern in the first image or the second image to be displayedmay be changed in accordance with a distance between the image capturedevice and the display device.

According to an aspect, an image pattern with a size appropriate forcalibration can be displayed in accordance with the relative distancebetween the display device and the image capture device. The accuracy ofa calibration of a coordinate system of an image capture device and acoordinate system of a robot arm based on a captured image depends onthe degree of accuracy to which the image capture device can identifythe size of the image pattern. Here, for example, an image patternincluded in a captured image is smaller when the distance from the imagecapture device to the display device is longer than when the distance isshorter. For this reason, it is favorable that an image pattern with anappropriate size is selected in accordance with the distance from theimage capture device to the display device. Accordingly, according to anaspect, an image is displayed based on an image pattern that can beaccurately identified by the image capture device, in accordance withthe relative distance between the image capture device and the displaydevice, and thus the calibration accuracy can be improved.

In an aspect, the step of changing the relative positions of the displaydevice and the image capture device may include a step of changing arelative angle between the display device and the image capture device,and at least data indicating a brightness of an image to be displayedmay differ between the first image data and the second image data.

According to an aspect, an image pattern with a brightness appropriatefor calibration can be displayed in accordance with the brightness orthe like that changes in accordance with the relative angle between thedisplay device and the image capture device. For example, when an imagepattern is displayed on the display device, there are cases wherebrightness conditions differ between measurement points between whichthe angle between the display device and the image capture devicediffers. In such cases, a situation where the image capture devicecannot identify the image displayed on the display device may occur.This situation often occurs when the relative angle between the displaydevice and the lighting is in a predetermined range. However, accordingto an aspect, brightness conditions for the image data aredifferentiated before and after the step of changing the relative anglebetween the display device and the image capture device. Accordingly, animage pattern with an appropriate brightness can be displayed inassociation with the brightness conditions that differ betweenmeasurement points, and it is thus possible to more reliably capture animage of the image displayed on the display device using the imagecapture device, and improve the calibration accuracy. Note that the stepof changing the relative angle between the display device and the imagecapture device may be performed simultaneously with the step of changingthe relative positions of the display device and the image capturedevice.

In an aspect, a relative positional orientation of the image capturedevice with respect to the display device during the step of acquiringthe first captured image data and a relative positional orientation ofthe image capture device with respect to the display device during thestep of acquiring the second captured image data may be the same.

According to an aspect, the quantity of information regarding thecaptured image data obtained at the same measurement point can beincreased, and thus the calibration accuracy is improved. Specifically,information includes a plurality of values to be substituted for eachterm in a computing equation for a calibration. By performingcomputation using the information including a plurality of values, andobtaining, for example, a weighted average for a plurality of calculatedtransformation matrices (values in the transformation matrices slightlydiffer from each other due to different patterns), a transformationmatrix in which errors have been further reduced can be obtained.

In an aspect, the display device may include a sensor for measuring abrightness, and a computing unit configured to change the first imagedata or the second image data based on a detection value from thesensor. During the step of causing the display device to display thefirst image and the step of causing the display device to display thesecond image, in the first image data or the second image data, abrightness of the first image or the second image to be displayed may bechanged in accordance with the detection value from the sensor formeasuring the brightness.

According to an aspect, an image pattern with an appropriate brightnesscan be displayed based on a detection value from the sensor included inthe display device. For example, to set an image pattern with anappropriate brightness that differs between measurement points, it isfavorable to detect the conditions under which light is applied to asubject whose image is to be captured by the image capture device.Accordingly, according to an aspect, an image pattern with anappropriate brightness is selected based on a detection value from thesensor included in the subject (display device) to which light from thelighting, natural light, or the like is applied. Thus, an image patternwith a brightness appropriate for the brightness conditions that differbetween the measurement points can be displayed, and the calibrationaccuracy can be improved. In an aspect, the display device may includean input unit configured to input information, and a computing unitconfigured to change the first image data or the second image data basedon the input information. During the step of causing the display deviceto display the first image and the step of causing the display device todisplay the second image, in the first image data or the second imagedata, a size and a brightness of a pattern in the first image or thesecond image to be displayed may be changed in accordance with the inputinformation.

According to an aspect, an image pattern with a size and brightnessappropriate for calibration can be displayed based on the informationinput from the input unit included in the display device. That is tosay, the image pattern is appropriately changed based on the informationthat is input, in advance, to the input unit in the display device, orthe information that is input every time an image is captured.Accordingly, it is possible to readily realize image pattern selection,and improve the calibration accuracy while reducing the number ofman-hours of operation.

In an aspect, the display device and the robot arm may be fixed to eachother. The display device may include a sensor for measuring an amountof movement. In the first image data or the second image data, a size ofa pattern in the first image or the second image to be displayed may bedetermined based on the amount of movement measured by the sensor.

According to an aspect, an image pattern with a size appropriate forcalibration can be displayed using the sensor included in the displaydevice. That is to say, since the display device can change the imagepattern to display in accordance with the amount by which the displaydevice moves, a simple configuration allows the image pattern to bechanged, and the calibration accuracy can be improved while reducing thenumber of man-hours of operation.

In an aspect, the display device and the robot arm may be fixed to eachother. The display device may include a sensor for measuring aninclination. Data indicating a brightness of an image to be displayedbased on at least one of the first image data and the second image datamay be determined based on a value of the inclination measured by thesensor. According to an aspect, an image pattern with a size appropriatefor calibration can be displayed using the sensor included in thedisplay device. That is to say, since the display device can change theimage pattern to display in accordance with the inclination thereof, asimple configuration allows the image pattern to be changed, and thecalibration accuracy can be improved while reducing the number ofman-hours of operation.

A calibration method according to an aspect is a method for calibratinga coordinate system of an image capture device and a coordinate systemof a robot arm in a robot system that includes a display device, theimage capture device, and the robot arm to which one of the displaydevice and the image capture device is fixed, the robot arm having adrive shaft. The method includes: a step of causing the display deviceto display a first image based on first image data; a step of capturingan image of the first image displayed on the display device andacquiring first captured image data, using the image capture device; anda step of calibrating a relative positional relationship between theimage capture device and the robot arm, using the first captured imagedata.

According to an aspect, a calibration of a relative positionalrelationship between the image capture device and the robot arm can beperformed without preparing calibration objects with a plurality ofpatterns printed thereon.

In an aspect, in the first image data, data indicating a pattern in animage to be displayed or data indicating a brightness of the image to bedisplayed may be determined in accordance with an image-capturingenvironment.

According to an aspect, a calibration can be performed using anappropriate image pattern in accordance with an image-capturingenvironment at the measurement point, and thus the calibration accuracycan be improved.

A program according to an aspect is a program for performing a methodfor calibrating a coordinate system of an image capture device and acoordinate system of a robot arm in a robot system that includes adisplay device, the image capture device, and the robot arm to which oneof the display device and the image capture device is fixed, the robotarm having a drive shaft, the program for causing a computer to perform:a step of causing the display device to display a first image based onfirst image data; a step of causing the image capture device to capturean image of the first image displayed on the display device to acquirefirst captured image data; a step of causing the display device todisplay a second image based on second image data different from thefirst image data; a step of causing the image capture device to capturean image of the second image displayed on the display device to acquiresecond captured image data; and a step of calibrating the coordinatesystem of the image capture device and the coordinate system of therobot arm using the first captured image data and the second capturedimage data.

According to an aspect, a program can be provided that is for causing acomputer to perform a calibration of a relative positional relationshipbetween the image capture device and the robot arm, without preparingcalibration objects with a plurality of patterns printed thereon. Sincea calibration can be performed using a plurality of captured images indifferent forms of display, the calibration accuracy can be improved.

In an aspect, data indicating a pattern in an image to be displayed ordata indicating a brightness of the image to be displayed may differbetween the first image data and the second image data.

According to an aspect, an image pattern with a brightness appropriatefor calibration can be displayed in accordance with the brightness ofthe surroundings or the like. For example, there are cases wherebrightness conditions differ between measurement points for measuringthe image using the image capture device, in accordance with an angle,illuminance, or the like of the lighting, and cases where brightnessconditions vary even at the same measurement point. Accordingly,according to an aspect, an appropriate image pattern can be displayed inaccordance with different brightness conditions, and thus thecalibration accuracy can be improved.

Furthermore, according to an aspect, the pattern in the image to bedisplayed can be differentiated, and thus the calibration accuracy canbe improved. That is to say, since a calibration can be performed usinga plurality of captured images of different patterns, the calibrationaccuracy can be improved. In addition, since the pattern can bedifferentiated in accordance with an environment in which an image is tobe captured, the calibration accuracy can be improved.

In an aspect, each of the first image data and the second image data maybe data indicating a pattern in an image to be displayed, and dataregarding a color of at least one pixel in the image to be displayed onthe display device may differ between the first image data and thesecond image data.

According to an aspect, the information of the captured image data canbe increased, and the calibration accuracy can be improved.

A calibration system according to an aspect includes: a display deviceconfigured to display a first image based on first image data, anddisplay a second image based on second image data different from thefirst image data; an image capture device configured to capture an imageof each of the first image and the second image, and acquire firstcaptured image data and second captured image data; a robot arm to whichone of the display device and the image capture device is fixed, therobot arm having a drive shaft; and a computing unit configured toperform computation to calibrate a coordinate system of the imagecapture device and a coordinate system of the robot arm, using the firstcaptured image data and the second captured image data.

According to an aspect, a calibration of a relative positionalrelationship between the image capture device and the robot arm can beperformed without preparing calibration objects with a plurality ofpatterns printed thereon. Since a calibration can be performed using aplurality of captured images in different forms of display, thecalibration accuracy can be improved.

In an aspect, data indicating a pattern in an image to be displayed ordata indicating a brightness of the image to be displayed may differbetween the first image data and the second image data.

According to an aspect, an image pattern with a brightness appropriatefor calibration can be displayed in accordance with the brightness ofthe surroundings or the like. For example, there are cases wherebrightness conditions differ between measurement points for measuringthe image using the image capture device, in accordance with an angle,illuminance, or the like of the lighting, and cases where brightnessconditions vary even at the same measurement point. Accordingly,according to an aspect, an appropriate image pattern can be displayed inaccordance with different brightness conditions, and thus thecalibration accuracy can be improved.

Furthermore, according to an aspect, the pattern in the image to bedisplayed can be differentiated, and thus the calibration accuracy canbe improved. That is to say, since a calibration can be performed usinga plurality of captured images of different patterns, the calibrationaccuracy can be improved. In addition, since the pattern can bedifferentiated in accordance with an environment in which an image is tobe captured, the calibration accuracy can be improved.

In an aspect, each of the first image data and the second image data maybe data indicating a pattern in an image to be displayed, and dataregarding a color of at least one pixel in the image to be displayed onthe display device may differ between the first image data and thesecond image data

According to an aspect, the information of the captured image data canbe increased, and the calibration accuracy can be improved.

One or more aspects can provide a calibration method and system, and aprogram that enable a calibration of a relative positional relationshipbetween an image capture device and a robot arm, without preparingcalibration objects with a plurality of patterns printed thereon.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram illustrating a calibration system.

FIG. 2 is a schematic diagram illustrating a calibration method in acase where a display 22 and a robot are fixed to each other.

FIG. 3 is a diagram illustrating an initial setting procedure for adisplay device.

FIGS. 4A to 4D are diagrams illustrating examples of calibrationobjects.

FIG. 5 is a flowchart illustrating a calibration method.

FIG. 6 is a schematic diagram illustrating a calibration method in acase where an image sensor and a robot are fixed to each other.

FIG. 7 is a diagram illustrating a screen for inputting relativepositional orientations of an image sensor and a robot.

DETAILED DESCRIPTION

Hereinafter, embodiments will be described with reference to theattached drawings (note that, in the diagrams, portions assigned thesame reference signs have the same or similar configurations). Thecalibration method that will be described below in the embodiments isfor calibrating a coordinate system of an image capture device and acoordinate system of a robot arm in a robot system that includes adisplay device, the image capture device, and the robot arm to which oneof the display device and the image capture device is fixed. Acalibration pattern that is based on image data is displayed on thedisplay device.

In the calibration method described in any of the following embodiments,the display device has a display for displaying a plurality of imagepatterns in which at least one of the calibration pattern and thebrightness of the image when the calibration pattern is displayeddiffers.

In the calibration method described in any of the following embodiments,the display device is configured to change the image pattern and changethe form of display for the image to be displayed on the display, inaccordance with an image-capturing environment. “Image-capturingenvironment” refers to an environment that is due to a configuration ofthe robot system, and includes, for example, a relative distance betweenthe image capture device and the display and relative orientationsthereof, which serve as relationships between the display and the imagecapture device. Also, “image-capturing environment” refers to anenvironment around the robot system, and is, for example, the brightnessof the surroundings when the image capture device captures an image ofan image pattern displayed on the display. With this configuration, thecalibration can be performed using an appropriate image pattern inaccordance with a change in the image-capturing environment, andaccordingly, the calibration accuracy can be improved.

In the calibration method described in any of the following embodiments,the display device may determine the image pattern to be displayed onthe display at a measurement point at which the image capture devicecaptures an image of the display, in accordance with the image-capturingenvironment. With this configuration, the calibration can be performedusing an appropriate image pattern in accordance with theimage-capturing environment at the measurement point, and accordingly,the calibration accuracy can be improved.

In the calibration method described in any of the following embodiments,the display device is configured to change the image pattern to bedisplayed on the display at a measurement point at which the imagecapture device captures an image of the display, and displays aplurality of images in different forms of display, on the display. Withthis configuration, the quantity of information regarding the image dataacquired at a measurement point can be increased, and thus thecalibration accuracy can be improved.

First Embodiment

FIG. 1 is a functional block diagram of a calibration system 10according to a first embodiment. This calibration system 10 has a robotR, and a robot controller 1 that has a robot control unit 12 forcontrolling the robot R. The calibration system 10 also includes animage sensor S, which serves as an image capture device. The robotcontroller 1 is configured to control operations of the robot R based onimages captured by the image sensor S. In an embodiment, the system thatincludes the robot R, the robot controller 1, and the image sensor S iscalled a robot system. The calibration system 10 includes an imagesensor controller 2 that has a sensor control unit 14 for controllingthe image sensor S, an image processing unit 16 for processing imagescaptured by the image sensor S, and a parameter calculation unit 18(computing unit) for calculating parameters of a calibration between therobot R and the image sensor S. The calibration system 10 also includesa display device D that has a display 22 and a display control unit 20for displaying images on this display 22. The control units, processingunit, and calculation unit are realized by a later-described hardwareprocessor executing predetermined programs. The control units,processing unit, and calculation unit can exchange data with each othervia wired or wireless communication.

The robot R is a multi-axis articulated robot arm R, which includes abase fixed to a floor surface, a plurality of joints that function asmovable shafts, and a plurality of links that rotate together with thejoints. The links are movably connected to each other via correspondingjoints. A link at a leading end of the robot arm R can be connected toan end effector E. In an embodiment, the robot arm R is a verticalarticulated robot that has six degrees of freedom and in which the baseand the plurality of links are connected in series via the joints.

The robot control unit 12 is arranged in a robot controller that has astorage device that stores a control program for controlling the robotR, and a processor for outputting a control signal for driving aservomotor provided in each joint of the robot R to rotate theservomotor in accordance with this control program.

The image sensor S is an image capture device for capturing an image toacquire captured image data, and is constituted by a CCD camera, forexample.

The sensor control unit 14 outputs a control signal for controlling theimage sensor S. The image processing unit 16 performs image processingon the captured image data acquired by the image sensor S, and outputsthe image-processed captured image data to the parameter calculationunit 18. The sensor control unit 14 has a storage device that stores aprogram for computing and outputting a control signal for controllingthe image sensor S, and a processor for performing computing processingin accordance with the program. The image processing unit 16 has astorage device that stores a program for performing image processing onthe captured image data and outputting the processing results to theparameter calculation unit 18, and a processor for performing computingprocessing in accordance with the program. Note that the sensor controlunit 14 and the image processing unit 16 may be configured integrallyand have a shared storage device and processor. The sensor control unit14 and the image processing unit 16 may be configured integrally withthe later-described parameter calculation unit 18 and have a sharedstorage device and processor.

The parameter calculation unit 18 is constituted by a computer programfor performing the calibration method according to an embodiment, astorage device that stores this computer program, data being processed,later-described image data, and so on, and a processor that performsvarious kinds of computing processing in accordance with the computerprogram.

A processor that is used in common for computing processing of each ofthe robot control unit 12, the sensor control unit 14, the imageprocessing unit 16, the parameter calculation unit 18, and thelater-described display control unit 20, or computing processing of atleast two units selected from thereamong may be a CPU (CentralProcessing Unit), a GPU (Graphic Processing Unit), an FPGA (FieldProgrammable Gate Array), a DSP (Digital Signal Processor), and an ASIC(Application Specific Integrated Circuit), together with a CPU orsolely. A storage device that is used in common for computing processingof each of the robot control unit 12, the sensor control unit 14, theimage processing unit 16, the parameter calculation unit 18, and thelater-described display control unit 20, or computing processing of atleast two units selected from thereamong has a nonvolatile storagedevice such as an HDD or an SSD, and a volatile storage device such as aDRAM or an SRAM.

In an embodiment, the parameter calculation unit 18 is provided in theimage sensor controller 2 (on the image sensor S side), and computingprocessing required for the calibration method according to anembodiment is performed by the processor provided in the parametercalculation unit 18. By thus using a processor other than the processorin the robot control unit 12, the load of the computing processingrequired for command processing for controlling the robot R andcalculation of calibration parameters can be distributed.

In another embodiment, at least some of the robot control unit 12, theimage processing unit 16, and the parameter calculation unit 18 may berealized by using shared hardware. For example, a calibration may beperformed using the processor in the robot controller, or the controlprogram for controlling the robot R and the program for performing acalibration may be stored in a shared storage device. In this regard, atleast two of the robot controller 1, the image sensor controller 2, andthe display device D may be configured integrally and share the storagedevice for storing the programs and the processor for performingcomputing processing.

The display control unit 20 generates image data for displaying variouscalibration patterns on the display 22, and outputs the generated imagedata to the display 22. The display 22 is constituted by, for example, aliquid-crystal display and so on, and displays an image based on theimage data received from the display control unit 20. For example, acommercially-available tablet, smartphone, or the like can be used ashardware of the display device D constituted by the display control unit20 and the display 22, but the display device D is not limited theretoand may alternatively be any other kind of liquid-crystal display deviceor organic EL display device. In an embodiment, the display control unit20 and the display 22 are integrally expressed as the display device D.However, the display control unit 20 and the display 22 may beseparately provided, rather than being integrated to serve as thedisplay device D. In this case, the display 22 may include a storagedevice that stores a computer program for performing display processingor the like, data being processed, and so on, and a processor thatperforms various kinds of computing processing in accordance with thecomputer program.

Note that, in an embodiment, the image data is stored in the storagedevice provided in the display control unit 20. The image data includesinformation regarding an image pattern that forms an image to bedisplayed on the display device D. The image pattern includes acalibration pattern, which is a pattern of an image to be displayed onthe display device D, and information for determining the brightness ofthe image to be displayed on the display device D. The mode of thecalibration pattern will be described later.

FIG. 2 is a schematic diagram illustrating an example of performing acalibration with the display device D (display 22) fixed to the robot R.An image pattern can be displayed with various relative angles andrelative positions of the display device D with respect to the imagecapture device S by moving the robot arm of the robot R while grippingthe display device D. The display device D is gripped and fixed by theend effector E connected at the leading end of the robot R. Meanwhile,the image capture device S is fixed to a ceiling of a factory or thelike, for example. Note that the method of fixing the display device Dto the robot R is not limited to gripping using the end effector E. Forexample, the display device D may be fixed to the link at the leadingend of the robot arm R via a connection structure such as a fixing jig.Alternatively, the display device D may be fixed to a link thatconstitutes the robot arm R via a predetermined connection structure. Inthe case of fixing the display device D to a link that constitutes therobot arm R, it is favorable that the display device D is fixed to thedistal-most link of the robot arm R.

FIG. 3 shows an example of an initial setting procedure for the displaydevice D. Initially, the type of calibration pattern to be displayed onthe display 22 is determined (S31). FIGS. 4A to 4D show examples ofcalibration patterns. FIG. 4A shows a dot image pattern constituted by asquare frame line and a total of 49 (7 rows by 7 columns) black dotsthat are arranged at regular intervals in this frame line. Meanwhile,one of the corners is painted in a triangular shape to specify thedirection of the square. That is to say, the calibration pattern shownin FIG. 4A is a pattern that includes black dots arranged atpredetermined intervals in the frame line, and a polygonal mark markerfor specifying the direction of the pattern. FIG. 4B shows imagepatterns of an AR marker (left) and a two-dimensional barcode (right).FIG. 4C shows an example of a checkerboard pattern (an image pattern inwhich black squares and white squares are arranged alternately), andFIG. 4D shows an example of an image pattern constituted by triangles.Various image patterns can be used in accordance with the positionalrelationship between the robot R and the image sensor S, the purpose ofcalibration, required accuracy, or the like. For example, the number andsize of black dots in the aforementioned dot pattern can be setarbitrarily. In an embodiment, the dot image pattern is selected. Thedisplay control unit 20 is configured to output image data fordisplaying the selected image pattern to the display 22.

Note that constituent elements of the calibration pattern included ineach image pattern are pixels, and a pixel is the minimum unit ofinformation regarding a color to be displayed on the display 22. Ifblack and white of at least one pixel are reversed, image patterns aredealt with as different ones. For this reason, two image patterns thathave similar shapes but have different sizes are different imagepatterns.

Next, the size of the calibration pattern is determined (S32). In anembodiment, the size of the pattern is determined based on the relativedistance between the display 22 and the image sensor S that iscalculated as follows.

First, the image sensor S is fixed to a ceiling or the like of a factoryin which the robot R is installed. Accordingly, the position coordinatesof the image sensor S are known. Similarly, the origin coordinates (basecoordinates), which is obtained with the base of the robot R serving asa reference, are also known. For this reason, the relative positions ofthese coordinates and the relative distance therebetween are obtained,in advance, with an error of several centimeters by means of aninstallation design drawing or actual measurement, for example.

Next, the leading end coordinates of the robot arm R are obtained.Specifically, the leading end coordinates are obtained by means ofkinetics calculation based on the rotation angle of each movable shaftand length data (shape data) of each link, with the origin coordinatesof the robot arm serving as a reference. Initially, the robot controlunit 12 acquires information regarding the rotation angle of eachmovable shaft, and outputs the acquired information to the parametercalculation unit 18. The parameter calculation unit 18 calculates theleading end coordinates based on the received rotation angleinformation, as well as the length data of each link and the origincoordinates of the robot R that are known.

Next, the coordinates of the display 22 (calibration object) areobtained. Specifically, the coordinates of the display 22 are obtainedbased on the known shape data (length data) of the display 22, with theleading end coordinates of the robot arm R serving as a reference. Ifthe position and orientation of the display 22 (calibration object)relative to the leading end coordinate system of the robot arm R arechangeable, the coordinates of the display 22 are obtained based notonly on the shape data of the display 22, but also on a changed positionand orientation. Next, the coordinates of the display 22 are obtainedwith the origin coordinates of the robot arm R serving as a reference,based on the coordinates of the display 22 relative to the leading endcoordinate system of the robot arm R.

Then, the relative distance between the display 22 and the image sensorS is calculated based on the calculated coordinates of the display 22and position coordinates of the image sensor S.

Note that the size of the later-described image pattern may be changedusing the relative distance (second relative distance) between theleading end coordinates of the robot arm R and the image sensor S, inplace of the relative distance (first relative distance) between thedisplay 22 and the image sensor S. Since the display 22 (calibrationobject) is smaller than the robot arm R, and the positional orientationof the display 22 varies in accordance with operations of the robot armR at the leading end coordinates, a change in the distance between thedisplay 22 and the image sensor S can also be addressed even if thesecond relative distance is used in place of the first relativedistance, and furthermore, the amount of computation can be reduced.

The display control unit 20 generates image data in which the size ofthe image pattern selected in step S31 has been adjusted based on therelative distance calculated by the parameter calculation unit 18, andoutputs the generated image data to the display 22. The display 22displays a predetermined image pattern based on the received image data.

Note that the method of calculating the relative distance between thedisplay 22 and the image sensor S is not limited to the above-describedmethod. For example, the relative distance may be calculated based on achange in the size of the pattern included in the calibration patternimage captured by the image sensor S. For example, an interval (firstinterval) between a first pattern element and a second pattern elementthat are predetermined and included in the calibration pattern isobtained in advance, and a relative distance (third relative distance)can be calculated based on information regarding the first interval,information indicating an interval (second interval) between the firstpattern element and the second pattern element extracted from a capturedimage, and sensor parameters of the image sensor S. This configurationmakes it possible to calculate the relative distance between the imagesensor S and the display 22 based on the size of the calibration patternin a captured image, the size changing in accordance with the distancebetween the display 22 and the image sensor S, and to vary the size theimage pattern to be displayed, in accordance with the calculatedrelative distance.

As described above, the size of the later-described image pattern can bevaried in accordance with a change in the leading end coordinates of therobot arm by using any of the first relative distance, the secondrelative distance, and the third relative distance.

Thus, in the calibration method according to an embodiment, the size ofthe image pattern to be displayed on the display 22 is determined inaccordance with the relative distance between the display 22 and theimage sensor S. In an embodiment, the size of the image pattern to bedisplayed (the area of a region enclosed by the square frame line) ispredetermined so as to substantially occupy approximately half the imagecapture region of the image sensor S.

Here, the size of an image pattern means the area of a portion enclosedby an outline of the image pattern, and can be expressed with the numberof pixels in the display 22, for example. Note that the size of an imagepattern may mean the size (area) of a pattern element that is presentwithin the outline of the image pattern. For example, in the calibrationpattern shown in FIG. 4A, the size of the image pattern may indicate thesize of each of the black dots that are present within the frame line ofthe calibration pattern, and the size of the polygonal mark marker.Also, for example, in the calibration patterns shown in FIGS. 4B to 4D,the size of the image pattern may indicate the size of a pattern elementthat is present within the frame line of each of these calibrationpatterns. The size of a pattern element that is present within the frameline of each of these image patterns may be changed in accordance with achange in the size of the outline of the image pattern, or may bechanged regardless of a change in the size of the outline of the imagepattern.

A specific calibration method will be described below using thedrawings. In an embodiment, the image pattern displayed as a calibrationpattern on the display 22 changes in accordance with variousorientations that the robot arm R assumes.

FIG. 5 is a flowchart of the calibration method according to anembodiment.

Initially, the robot arm R assumes a predetermined orientationcorresponding to a measurement point at which an image is to becaptured, based on a control signal from the robot control unit 12(S41). At this time, the orientation of the robot arm R is predeterminedso that an entire display screen of the display device D is contained inan image capture region that can be captured by the image sensor S.

Next, the relative distance between the leading end coordinates of therobot arm R assuming this orientation (the coordinates of the display22) and the image sensor S is calculated using the above-describedmethod by the parameter calculation unit 18, and the calculated relativedistance is output to the display control unit 20. Subsequently, imagedata for displaying a dot pattern with a size that is based on therelative distance is generated by the display control unit 20, and isoutput to the display 22. Simultaneously, the sensor control unit 14causes the image sensor S to capture an image of the image patterndisplayed on the display 22, and acquires the captured image data (S42).

Next, whether or not the number of times that an image is captured hasreached a predetermined number of times (N times) is determined by theparameter calculation unit 18 (S43). If not, steps S42 and S43 arerepeated. Specifically, processing to move to a measurement pointdifferent from the previous measurement point by causing the robot arm Rto assume an orientation different from the previous one, processing tocalculate the relative distance between the leading end coordinates (thecoordinates of the display 22) at this time and the image sensor S,processing to generate image data for displaying a dot pattern with acorresponding size, and processing to cause the image sensor S tocapture an image of the image pattern displayed on the display 22 toacquire the captured image data are repeatedly performed.

Here, a smaller image pattern is displayed when the calculated relativedistance is short than when the relative distance is long, and a largerimage pattern is displayed when the relative distance is long than whenthe relative distance is short. For example, if the calculated relativedistance is 1.2 times greater than that calculated at the previousmeasurement point, image data is generated so that each side of thesquare outer frame of the dot pattern and the diameter of each dot inthe outer frame are 1.2 times greater than those at the previous time,and is displayed on the display 22. On the other hand, if the calculatedrelative distance is 0.8 times the relative distance calculated at theprevious measurement point, image data is generated so that each side ofthe outer frame and the diameter of each dot in the outer frame are 0.8times smaller than those at the previous time, and is displayed on thedisplay 22. This configuration makes it possible to keep the size of theimage pattern relative to the image capture region of the image sensor Sto a predetermined value or more, and to thus improve the calibrationaccuracy.

Note that not only dot image patterns with different sizes but alsoimage pattern with different forms may be displayed, and images thereofmay be captured. For example, the calibration accuracy can be improvedby displaying image patterns in which black and white are reversed andacquiring captured image data for the respective images. That is to say,in at least two image patterns to be displayed, color informationregarding at least one of the pixels corresponding to the display deviceD may be differentiated between a first image pattern and a second imagepattern. More specifically, in the first image pattern and the secondimage pattern, white and black may be reversed in at least one of thepixels corresponding to the display device D. This configuration makesit possible to increase the quantity of acquirable information regardingcaptured images and improve the calibration accuracy.

The size of the image pattern displayed on the display 22 may becontinuously changed while moving the robot arm R to acquire capturedimage data of the image patterns of the respective sizes. For example,the image pattern displayed on the display 22 may be reduced (orincreased) by a fixed size while controlling the robot arm R so that thedisplay 22 approaches (or moves away from) the image sensor S at a fixedspeed.

If the number of times that an image is captured has reached thepredetermined number of times (N times), and images of N image patternsdisplayed for N orientations have been captured, the positionalorientation of the calibration object (the display 22 on which thecalibration pattern is displayed) is calculated based on N sets ofcaptured image data by the parameter calculation unit 18 (S44).

Next, relative positional orientations of the robot and the sensor areobtained by the parameter calculation unit 18 based on the calculatedpositional orientation of the calibration object, and the positionalorientation of the tool (S45). Specifically, a relative positionalrelationship between the robot and the sensor is acquired based on thefollowing equation.^(cam) H _(cal)=^(cam) H _(base)*^(base) H _(tool)*^(tool) H _(cal)

Here, ^(cam)H_(cal) in the left-hand side denotes a transformationmatrix (including a rotation matrix and a translation vector) thatindicates the positional orientation of the calibration object based onthe coordinate system of the image capture device (camera), and iscalculated based on the captured image data, as in the above-describedsteps.

^(base)H_(tool) denotes a transformation matrix that indicates thepositional orientation of the leading end of the robot arm R that isbased on the origin coordinate system thereof, and is calculated, asdescribed above, by means of kinetics calculation based on the rotationangle of each movable shaft constituting the orientation of the robotarm R when the captured image data was acquired, and the known lengthdata of each link.

^(tool)H_(cal) denotes a transformation matrix indicating the positionalorientation of the calibration object based on the leading endcoordinate system of the robot arm R, and can be acquired, in advanced,based on the size of the end effector E and the display device D, forexample. If the position and orientation of the calibration objectrelative to the leading end coordinate system of the robot arm R do notchange during at least two times of image capturing, the computation ofthe transformation matrix expressed as ^(tool)H_(cal) can be omitted.The cases where the computation of the transformation matrix expressedas ^(tool)H_(cal) can be omitted include, for example, a case whereimage capturing is performed at least twice under a condition that thecalibration object is fixed to the leading end of the robot arm R. Inthis case, when computation for a calibration is performed using twosets of captured image data acquired in association with each time ofimage capturing, a condition that the positional orientation of thecalibration object relative to the leading end coordinate system of therobot arm R does not change can be used. Accordingly, the computation ofthe transformation matrix expressed as ^(tool)H_(cal) can be omitted.

Accordingly, a transformation matrix ^(cam)H_(base) that indicates thepositional orientation of the robot R at the origin coordinates seenfrom the coordinate system of the image sensor S, or an inverse matrix^(base)H_(cam) of this transformation matrix can be acquired asinformation indicating a relative positional relationship between therobot R and the image sensor S.

Through the above steps, a calibration is complete in which the relativepositional relationship between the image sensor S, which is an imagecapture device, and the robot arm R, which is a robot, is acquired.

Modifications

Modifications of the above-described first embodiment will be describedbelow. Redundant descriptions will be omitted, and different points willbe described.

In a first embodiment, the robot arm R and the display device D arefixed to each other, and captured image data required for a calibrationis acquired while moving the display device D using the robot arm R.This modification will describe a method in which the robot arm R andthe image sensor S are fixed to each other, and captured image datarequired for a calibration is acquired while moving the image sensor Susing the robot arm R.

First, the image sensor S is fixed to the robot arm R. Somecommercially-available robot arms R are originally provided with animage sensor S. Such a robot arm R can perform a calibration using theimage sensor S thereof. For a robot arm R that is not originallyprovided with the image sensor S, the image sensor S needs to be fixedto a link at the leading end, for example, using a jig or the like.Meanwhile, the display 22 is fixed at a predetermined position.

FIG. 6 is a schematic diagram illustrating a state where an image of animage pattern displayed on the display 22 is captured using the robotarm R with the image sensor S fixed thereto.

A specific calibration method will be described below using thedrawings. In this modification as well, the calibration pattern includedin the image pattern displayed on the display 22 changes in accordancewith various orientations that the robot arm R assumes. The flowchart inFIG. 5 can be used as that of the calibration method according to thismodification. The robot arm R assumes a predetermined orientation sothat an entire display screen of the display 22 is contained in an imagecapture region that can be captured by the image sensor S (S41).

Next, the sensor control unit 14 causes the image sensor S to capture animage of the image pattern displayed on the display 22, and acquirescaptured image data (S42). However, the size of the image pattern can bedetermined using a method other than that in a first embodiment. Forexample, the size of the image pattern can be determined based on therelative distance between the leading end coordinates of the robot arm Rand the position coordinates of the fixed display 22.

Next, whether or not the number of times that an image is captured hasreached a predetermined number of times (N times) is determined by theparameter calculation unit 18 (S43). If not, steps S42 and S43 arerepeated. Specifically, the image sensor S captures images of variousimage patterns displayed on the display 22 to acquire the captured imagedata while the robot arm R is assuming various orientations.

If the number of times that an image is captured has reached apredetermined number of times (N times), the positional orientation ofthe calibration object (the display 22 on which the calibration patternis displayed) is calculated based on the captured image data by theparameter calculation unit 18 (S44).

Then, the relative positional orientations of the robot and the sensorare obtained by the parameter calculation unit 18 based on thecalculated positional orientation of the calibration object and thepositional orientation of the tool (S45). Specifically, a relativepositional relationship between the robot and the sensor is acquiredbased on the following equation.^(cam) H _(cal)=^(cam) H _(tool)*^(tool) H _(base)*^(base) H _(cal)

Here, ^(cam)H_(cal) in the left-hand side denotes a transformationmatrix (including a rotation matrix and a translation vector) thatindicates the positional orientation of the calibration object based onthe coordinate system of the camera, and is calculated based on thecaptured image data, as in the above-described steps.

^(tool)H_(base) denotes an inverse matrix of the transformation matrixthat indicates the positional orientation at the leading end that isbased on the origin coordinate system of the robot arm R in a firstembodiment, and is calculated by means of kinetics calculation based onthe rotation angle of each movable shaft constituting the orientation ofthe robot arm R when the captured image data was acquired, and the knownlength data of each link.

^(base)H_(cal) a transformation matrix that indicates the positionalorientation of the calibration object that is based on the origincoordinate system of the robot arm R, and can be acquired in advance,based on the position coordinates of the display device 22, the size ofthe display device D, and the like. If the position and orientation ofthe calibration object relative to the origin coordinate system of therobot arm R do not change during at least two times of image capturing,the computation of the transformation matrix expressed as ^(base)H_(cal)can be omitted. The cases where the computation of the transformationmatrix expressed as ^(base)H_(cal) can be omitted include, for example,a case where the robot arm R is operated, images of the calibrationobject can be captured using the image sensor S before and after theorientation of the robot arm R is changed, and the location to which thecalibration object is fixed does not need to be changed.

Accordingly, a transformation matrix ^(cam)H_(tool) that indicates thepositional orientation of the robot R at the leading end coordinatesseen from the coordinate system of the image sensor S, or an inversematrix ^(tool)H_(cam) of this transformation matrix can be acquired asinformation indicating a relative positional relationship between therobot R and the image sensor S.

Through the above steps, a calibration is complete in which the relativepositional relationship between the image sensor S, which is an imagecapture device, and the leading end coordinates of the robot arm R isacquired.

As described above, with the calibration method based on an embodimentand the modification thereof, a relative positional relationship betweenthe image sensor S, which is an image capture device, and the robot armR can be acquired without preparing calibration objects with a pluralityof patterns printed thereon.

Note that the size of the square frame of the dot image pattern, thenumber of dots therein, and the interval between dots can be setarbitrarily. Various image patterns may be combined to be used in acalibration in accordance with the purpose of calibration, or the like.

The calibration system 10 may also include an output unit for outputtinga relative positional relationship between the robot R and the imagesensor S.

The size of the image pattern may not be determined based on therelative distance between the display 22 and the leading end coordinatesof the robot arm R, and may alternatively be determined based on otherindexes. For example, the size of the image pattern may be determined inaccordance with the distance between a predetermined point and theleading end coordinates of the robot R.

“Leading end coordinates” is intended to include not only thecoordinates of the link located at the tip, of the links constitutingthe robot arm R, but also the coordinates of a point that is locatedforward of the tip of (i.e. located at a terminal of) the robot arm,such as the leading end or intermediate coordinates of a tool (endeffector).

Second Embodiment

A second embodiment will describe points different from a firstembodiment, while omitting descriptions of the same features.

In a first embodiment, the size of the image pattern is changed based onthe relative distance between the image sensor S and the leading end ofthe robot arm R. In an embodiment, the size of the calibration pattern(image pattern) is changed based on the amount of blur in thecalibration pattern whose image is captured by the image sensor S.

First, a method of estimating the distance based on the amount of blurwill be described. If the focal length of an imaging plane lens of theimage sensor S is denoted as f, and the distance between the imagesensor S and a light source A is denoted as u, the distance v at whichthe light source A is brought into focus satisfies a relationshipexpressed as f⁻¹=u⁻¹+v⁻¹. Accordingly, if the distance r from theimaging plane lens to the imaging plane satisfies r=v, an in-focus imagecan be obtained. On the other hand, if r≠v, when the aperture of theimage sensor S (camera) is denoted as p, the light source A is projectedas a circle having a diameter q=p*|v−r|/v on the imaging plane, and ablur with an amount of blur q occurs.

Accordingly, the amount of blur q can be obtained by preparing, as animage pattern, image data that includes a point, capturing an image ofthe image pattern, and performing image recognition on captured imagedata to check the diameter of the point appearing as a circle in thecaptured image. Then, the distance u between the image sensor S and thecalibration object can be estimated based on p, r, and f, which areknown, and the above equation.

By changing the size of the image pattern based on the distance u, thesize of the calibration pattern to be displayed can be increased even ifthe image sensor S and the display 22 move away from each other. Also,even if the image sensor S and the display 22 move closer to each other,the entire calibration pattern can be displayed within the image captureregion of the image sensor S by reducing the size of the calibrationpattern to be displayed.

Furthermore, since computing processing to calculate the leading endcoordinates of the robot R and so on can be omitted, the load on theprocessors can be reduced.

Note that, instead of the distance estimation based on the amount ofblur, a configuration may alternatively be employed in which a marker isdisplayed as an image pattern displayed on the image sensor, and thedistance between a plurality of feature points in the marker iscalculated by means of image recognition, and the size of the imagepattern may be changed based on this distance (or the amount of changein the distance between the feature points according to a change in theorientation).

Third Embodiment

(Halation) A third embodiment will describe points different from theabove-described embodiments, while omitting the same features.

A calibration system according to an embodiment includes an illuminancesensor. The brightness of the image pattern displayed on the display 22is differentiated in accordance with the value of the ambientilluminance obtained by the illuminance sensor. It is favorable that theilluminance sensor is integrally provided with the display 22.Specifically, in the case of high illuminance (e.g. in the case where adisplay surface of the display 22 opposes lighting on the ceiling, andthus the light reflected off the display surface is received by theimage sensor 5), the display 22 is caused to display an image patternbased on image data with which the image pattern is relatively darklydisplayed compared with the case of low illuminance. In the case of lowilluminance (e.g. in the case where the display surface of the display22 faces toward a floor, and thus the lighting on the ceiling is hardlyreceived by the image sensor 5), the display 22 is caused to display animage pattern based on image data with which the image pattern isrelatively brightly displayed compared with the case of highilluminance. It is thereby possible to perform a calibration whilereducing the influence of a halation (a phenomenon in which theperiphery of a subject is blurred whitely and unclear due to anexcessively strong light beam) that is caused due to, for example, thelighting installed on the ceiling.

For example, degradation of the calibration accuracy due to a halationcan be suppressed without changing the relative distance between theimage sensor S and the display 22, by controlling the robot arm R so asto change the relative angle of the display 22 with respect to the imagesensor S and changing the brightness of the displayed image inaccordance with the output of an illuminometer.

If, for example, the position of the lighting is known, a configurationmay be employed in which image data that indicates the brightness isadjusted based on the positional orientation of the robot arm R, anestimated angle of the display surface of the display 22 relative to thefloor surface, or the like.

If a tablet-type mobile information processing terminal or a smartphoneis used as the display device D, an illuminance sensor that isoriginally provided in this terminal may be used.

In an embodiment, “image data” is intended to include not only the shapeof a displayed image but also an item whose brightness is to bedifferentiated, and includes not only data indicating a brightness thatis designated for each pixel, but also data indicating control data thatdefines the output of a backlight of the display 22, and the degree ofbrightness.

Fourth Embodiment

(Use of functions of display device, input unit, sensor etc.) A fourthembodiment will describe points different from the above-describedembodiments, while omitting the same features.

In the above-described embodiments, the processors are used to calculatethe relative distance between the image sensor S and the leading end ofthe robot arm R, and so on. However, by providing the display device Dwith various functions, the display device D can alternatively be usedin complicated computing processing to be performed using processors orthe like.

For example, as shown in FIG. 7, a configuration can be employed inwhich an input I/F (input means) is provided in the display device D,and the input of estimated values of relative positional orientations ofthe robot R and the image sensor S, and a measurement range and visualfield range of the image sensor S (or the input of a model number or thelike of the image sensor S that defines these values) are received. Inthis configuration, an image pattern with an appropriate size, shape,and brightness can be automatically displayed in accordance with theinput values. A configuration may also be employed in which the shapesof a plurality of calibration objects are selectably displayed in ascrollable manner on the display 22. Note that the mode of the input I/Fis not particularly limited to the mode shown in FIG. 7. For example,information that determines the content of an image pattern may be inputvia a communication I/F that is provided in the display device D forwired or wireless electrical communication with an external device.

A configuration may also be employed in which an acceleration sensor, aspeed sensor, or a tilt (gyroscope) sensor is provided in the displaydevice D or the robot arm R, and an image pattern with an appropriatesize, shape, and brightness is automatically displayed based onmeasurement data from the sensor indicating the amount by which therobot arm R has moved or the inclination of the robot arm R.

This configuration makes it possible to objectively select an imagepattern with an appropriate shape, size, or the like. In addition, aphysical configuration or the like for transmission and reception, i.e.for receiving data for calculating the relative distance or the likefrom the robot arm R is not needed, and image data for displaying animage pattern can be determined only with the display device D.Furthermore, in the case of using a mobile terminal (such as a tablet ora smartphone) as the display device D, an input means and varioussensors that are originally provided therein can be used.

Note that all of the embodiments may employ a configuration in whichimages are displayed based on different image patterns at the samemeasurement point, and computation for a calibration is performed basedon a plurality of captured images acquired at the same measurementpoint. Here, it is favorable that “different image patterns” are imagepatterns in which color information regarding at least one pixelcorresponding to the display device D differs. It is particularlyfavorable that “different image patterns” are image patterns in whichwhite and black are reversed in at least one pixel in the displaydevice. This configuration makes it possible to increase the quantity ofinformation regarding the captured images acquired at each measurementpoint, and improve the calibration accuracy. That is to say, valueinformation on which calibration computation is based can be increasedby displaying a plurality of types of image patterns in different modesat the same measurement point. Accordingly, with this configuration,computing processing for a calibration can be performed using aplurality of types of image patterns acquired at the same measurementpoint, and thus the calibration accuracy can be improved.

The above-described embodiments are for facilitate understanding of thepresent invention, and is not for interpreting the present invention ina limited manner. The elements provided in the embodiments, thearrangement, material, conditions, shape, size, and the like of thoseelements are not limited to those described as an example, and may bechanged as appropriate. Configurations described in differentembodiments may be partially replaced, or may be combined.

Note that, in this specification, “unit”, “means”, and “procedure” donot simply mean physical configurations, but also include the caseswhere processing performed by such “unit” and the like is realized bysoftware. Processing to be performed by one “unit” or the like ordevices may be performed by two or more physical configurations ordevices. Processing to be performed by two or more “units” or the likeor devices may be performed by one physical means or device.

The above embodiments may also be described, entirely or partially, asin the following Notes, but are not limited thereto.

(Note 1)

A method for calibrating a coordinate system of an image capture deviceand a coordinate system of a robot arm in a robot system that includes adisplay device, the image capture device, and the robot arm to which oneof the display device and the image capture device is fixed, the robotarm having a drive shaft, the method including:

a step of causing the display device to display a first image based onfirst image data;

a step of capturing an image of the first image displayed on the displaydevice and acquiring first captured image data, using the image capturedevice;

a step of causing the display device to display a second image based onsecond image data different from the first image data;

a step of capturing an image of the second image displayed on thedisplay device and acquiring second captured image data, using the imagecapture device; and

a step of calibrating the coordinate system of the image capture deviceand the coordinate system of the robot arm using the first capturedimage data and the second captured image data.

(Note 2)

A method for calibrating a coordinate system of an image capture deviceand a coordinate system of a robot arm in a robot system that includes adisplay device, the image capture device, and the robot arm to which oneof the display device and the image capture device is fixed, the robotarm having a drive shaft, the method including:

a step of causing the display device to display a first image based onfirst image data;

a step of capturing an image of the first image displayed on the displaydevice and acquiring first captured image data, using the image capturedevice; and

a step of calibrating the coordinate system of the image capture deviceand the coordinate system of the robot arm, using the first capturedimage data.

(Note 3)

A program for performing a method for calibrating a coordinate system ofan image capture device and a coordinate system of a robot arm in arobot system that includes a display device, the image capture device,and the robot arm to which one of the display device and the imagecapture device is fixed, the robot arm having a drive shaft,

the program for causing a computer to perform:

causing the display device to display a first image based on first imagedata;

causing the image capture device to capture an image of the first imagedisplayed on the display device to acquire first captured image data;

causing the display device to display a second image based on secondimage data different from the first image data;

causing the image capture device to capture an image of the second imagedisplayed on the display device to acquire second captured image data;and

calibrating the coordinate system of the image capture device and thecoordinate system of the robot arm using the first captured image dataand the second captured image data.

(Note 4)

A calibration system comprising:

a display device configured to display a first image based on firstimage data, and display a second image based on second image datadifferent from the first image data;

an image capture device configured to capture an image of each of thefirst image and the second image, and acquire first captured image dataand second captured image data;

a robot arm configured to change relative positions of the displaydevice and the image capture device, one of the display device and theimage capture device being fixed to the robot arm; and

a computing unit configured to perform computation to calibrate acoordinate system of the image capture device and a coordinate system ofthe robot arm, using the first captured image data and the secondcaptured image data.

The invention claimed is:
 1. A method for calibrating a coordinatesystem of an image capture device and a coordinate system of a robot armin a robot system that includes a display device, the image capturedevice, and the robot arm to which one of the display device and theimage capture device is fixed, the robot arm comprising a drive shaft,the method comprising: causing the display device to display a firstimage based on first image data; capturing an image of the displayedfirst image and acquiring first captured image data, using the imagecapture device; causing the display device to display a second imagebased on second image data different from the first image data;capturing an image of the displayed second image and acquiring secondcaptured image data, using the image capture device; and calibrating thecoordinate system of the image capture device and the coordinate systemof the robot arm using the first captured image data and the secondcaptured image data.
 2. The calibration method according to claim 1,wherein data indicating a pattern in an image to be displayed or dataindicating a brightness of the image to be displayed differs between thefirst image data and the second image data.
 3. The calibration methodaccording to claim 1, wherein each of the first image data and thesecond image data comprise data indicating a pattern in an image to bedisplayed, and data regarding a color of at least one pixel in the imageto be displayed differs between the first image data and the secondimage data.
 4. The calibration method according to claim 1, furthercomprising: changing relative positions of the display device and theimage capture device using the robot arm, after acquiring the firstcaptured image data, wherein acquiring the second captured image data isperformed after changing the relative positions.
 5. The calibrationmethod according to claim 1, wherein, while causing the display deviceto display the first image and causing the display device to display thesecond image, in the first image data or the second image data, a sizeof a pattern in the first image or the second image to be displayed ischanged in accordance with coordinates of a leading end of the robotarm.
 6. The calibration method according to claim 1, wherein, whilecausing the display device to display the first image and causing thedisplay device to display the second image, in the first image data orthe second image data, a size of a pattern in the first image or thesecond image to be displayed is changed in accordance with a distancebetween the image capture device and the display device.
 7. Thecalibration method according to claim 1, further comprising: changing arelative angle between the display device and the image capture deviceusing the robot arm, after acquiring the first captured image data,wherein at least data indicating a brightness of an image to bedisplayed differs between the first image data and the second imagedata.
 8. The calibration method according to claim 1, wherein a relativepositional orientation of the image capture device with respect to thedisplay device while acquiring the first captured image data and arelative positional orientation of the image capture device with respectto the display device while acquiring the second captured image data arethe same.
 9. The calibration method according to claim 1, wherein thedisplay device includes a sensor for measuring a brightness, and aprocessor configured with a program to perform operations comprisingoperation as a computing unit configured to change the first image dataor the second image data based on a detection value from the sensor, andwhile causing the display device to display the first image and causingthe display device to display the second image, in the first image dataor the second image data, a brightness of the first image or the secondimage to be displayed is changed in accordance with the detection valuefrom the sensor for measuring the brightness.
 10. The calibration methodaccording to claim 1, wherein the display device comprises a processorconfigured with a program to perform operations comprising: operation asan input unit configured to input information; and operation as acomputing unit configured to change the first image data or the secondimage data based on the input information, and while causing the displaydevice to display the first image and causing the display device todisplay the second image, in the first image data or the second imagedata, a size and a brightness of a pattern in the first image or thesecond image to be displayed are changed in accordance with the inputinformation.
 11. The calibration method according to claim 1, whereinthe display device and the robot arm are fixed to each other, thedisplay device includes a sensor for measuring an amount of movement,and in the first image data or the second image data, a size of apattern in the first image or the second image to be displayed isdetermined based on the amount of movement measured by the sensor. 12.The calibration method according to claim 1, wherein the display deviceand the robot arm are fixed to each other, the display device includes asensor for measuring an inclination, and data indicating a brightness ofan image to be displayed based on at least one of the first image dataand the second image data is determined based on a value of theinclination measured by the sensor.
 13. A method for calibrating acoordinate system of an image capture device and a coordinate system ofa robot arm in a robot system that includes a display device, the imagecapture device, and the robot arm to which one of the display device andthe image capture device is fixed, the robot arm comprising a driveshaft, the method comprising: causing the display device to display afirst image based on first image data; capturing an image of thedisplayed first image and acquiring first captured image data, using theimage capture device; causing the display device to display a secondimage based on second image data different from the first image data;capturing an image of the displayed second image and acquiring secondcaptured image data, using the image capture device; and calibrating arelative positional relationship between the image capture device andthe robot arm, using the first captured image data and the secondcaptured image data.
 14. The calibration method according to claim 13,wherein, in the first image data, data indicating a pattern in an imageto be displayed or data indicating a brightness of the image to bedisplayed is determined in accordance with an image-capturingenvironment.
 15. A non-transitory computer-readable storage recordingmedium storing a program for performing a method for calibrating acoordinate system of an image capture device and a coordinate system ofa robot arm in a robot system that includes a display device, the imagecapture device, and the robot arm to which one of the display device andthe image capture device is fixed, the robot arm comprising a driveshaft, the program causing a computer to perform operations comprising:causing the display device to display a first image based on first imagedata; causing the image capture device to capture an image of thedisplayed first image to acquire first captured image data; causing thedisplay device to display a second image based on second image datadifferent from the first image data; causing the image capture device tocapture an image of the displayed second image to acquire secondcaptured image data; and calibrating the coordinate system of the imagecapture device and the coordinate system of the robot arm using thefirst captured image data and the second captured image data.
 16. Thenon-transitory computer-readable recording medium storing the programaccording to claim 15, wherein data indicating a pattern in an image tobe displayed or data indicating a brightness of the image to bedisplayed differs between the first image data and the second imagedata.
 17. The non-transitory computer-readable recording medium storingthe program according to claim 15, wherein each of the first image dataand the second image data comprise data indicating a pattern in an imageto be displayed, and data regarding a color of at least one pixel in theimage to be displayed on the display device differs between the firstimage data and the second image data.
 18. A calibration systemcomprising: a display device configured to display a first image basedon first image data, and display a second image based on second imagedata different from the first image data; an image capture deviceconfigured to capture an image of each of the first image and the secondimage, and acquire first captured image data and second captured imagedata; a robot arm to which one of the display device and the imagecapture device is fixed, the robot arm comprising a drive shaft; and aprocessor configured with a program to perform operations comprisingoperation as a computing unit configured to perform computation tocalibrate a coordinate system of the image capture device and acoordinate system of the robot arm, using the first captured image dataand the second captured image data.
 19. The calibration system accordingto claim 18, wherein data indicating a pattern in an image to bedisplayed or data indicating a brightness of the image to be displayeddiffers between the first image data and the second image data.
 20. Thecalibration system according to claim 18, wherein each of the firstimage data and the second image data comprises data indicating a patternin an image to be displayed, and data regarding a color of at least onepixel in the image to be displayed on the display device differs betweenthe first image data and the second image data.